Tidal Culvert Data QA-QC

** TOTAL WORK IN PROGRESS **

This attempts to both outline the current status of the tidal culvert data and provide a task list for walking through the data collected to date and the data that are yet to be collected. Sort of a hybrid documentation of methods and to-do list.

library(knitr)
library(rmdformats)

## Global options
options(max.print="75")
opts_chunk$set(echo=FALSE,
                 cache=TRUE,
               prompt=FALSE,
               tidy=TRUE,
               comment=NA,
               message=FALSE,
               warning=FALSE)
opts_knit$set(width=75)

library(readr)
library(readxl)
library(tidyxl)
library(unpivotr)
library(tidyverse)
library(sf)
library(mapview)
library(rstudio)

keysheet <- read_excel("spreadsheets/key.xlsx")

LIculvert_GISpts <- st_read("M:/Projects/LI/Culvert_Assessment/data/Tidal Crossings/TidalCrossings_20180712.shp")

## Reading layer `TidalCrossings_20180712' from data source `M:\Projects\LI\Culvert_Assessment\data\Tidal Crossings\TidalCrossings_20180712.shp' using driver `ESRI Shapefile'
## Simple feature collection with 177 features and 16 fields
## geometry type:  POINT
## dimension:      XY
## bbox:           xmin: -1.797693e+308 ymin: -1.797693e+308 xmax: -8004521 ymax: 5034111
## epsg (SRID):    3857
## proj4string:    +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs

LIculvert_GISpts <- LIculvert_GISpts %>% mutate(crossingID = as.character(Tidal_ID))

cleanLIculvert_GISpts <- LIculvert_GISpts %>% filter(Latitude != 0)

source("functions/culvert_tidy.R")
source("functions/culvert_extract.R")    

LIculvertsAssessments <- culvert_tidy("../../Box Sync/Tidal Assessments/Culvert Excel Sheets") %>% 
    mutate(decoded = map(.x = tidycells, .f = ~decodeSheet(.x, keysheet)))  # this is where the sausage is made. 


LIculvertData <- LIculvertsAssessments %>% select(filenames, decoded) %>% unnest() %>% 
    select(filenames, dataName, values) %>% spread(key = dataName, value = values) %>% 
    select(filenames, crossingID, dateAssessed, observers, everything())  # organize the order of the columns.


# LIculvertData %>% summarize_all(funs(sum(is.na(.))))


if (typeof(LIculvertData$crossingID) == typeof(LIculvert_GISpts$crossingID)) {
    # Are the two columns the same datatype?
    matchedLIculverts <- LIculvertData %>% filter(crossingID %in% LIculvert_GISpts$crossingID) %>% 
        select(filenames, crossingID)
    missingLIculevrts <- LIculvertData %>% filter(!crossingID %in% LIculvert_GISpts$crossingID) %>% 
        select(filenames, crossingID)
}

Output

Here’s the output– I added an option to the table so that a local version can be saved- It’s a bit clumbsy here since it’s so wide (lots of variables) but you can scroll left/right to see all the columns- or click ‘Download’ to save a copy to open in Excel.

Missing CrossingID’s

The following list contains filenames of datasheets that are missing the CrossingID field. Presumably this is due to the lack of field assessments and/or Desktop assessments, perhaps due to these sites not landing on any priotity lists (for assessments). Cross reference this list with the prioirty crossings lists to see which to focus on first. Just a note that currently the only linkage between the field data sheets and the spatial data (the locations for culverts as derived from remotely sensed and groundtruthed data) are the CrossingID field. Care must be taken to ensure the names are entered correctly.

Mapped Culverts

The map below are locations (though some inaccuracies are evident…) of culverts with related ‘field datasheet’ info. Click a point to see details.

Prioity sites.

These are just color coded based on the Priority listed in the shapefile